Multi-agent Reinforcement Learning with Sparse Interactions by Negotiation and Knowledge Transfer

摘要

Reinforcement learning has significant applications for multi-agent systems,especially in unknown dynamic environments. However, most multi-agentreinforcement learning (MARL) algorithms suffer from such problems asexponential computation complexity in the joint state-action space, which makesit difficult to scale up to realistic multi-agent problems. In this paper, anovel algorithm named negotiation-based MARL with sparse interactions (NegoSI)is presented. In contrast to traditional sparse-interaction based MARLalgorithms, NegoSI adopts the equilibrium concept and makes it possible foragents to select the non-strict Equilibrium Dominating Strategy Profile(non-strict EDSP) or Meta equilibrium for their joint actions. The presentedNegoSI algorithm consists of four parts: the equilibrium-based framework forsparse interactions, the negotiation for the equilibrium set, the minimumvariance method for selecting one joint action and the knowledge transfer oflocal Q-values. In this integrated algorithm, three techniques, i.e., unsharedvalue functions, equilibrium solutions and sparse interactions are adopted toachieve privacy protection, better coordination and lower computationalcomplexity, respectively. To evaluate the performance of the presented NegoSIalgorithm, two groups of experiments are carried out regarding three criteria:steps of each episode (SEE), rewards of each episode (REE) and average runtime(AR). The first group of experiments is conducted using six grid world gamesand shows fast convergence and high scalability of the presented algorithm.Then in the second group of experiments NegoSI is applied to an intelligentwarehouse problem and simulated results demonstrate the effectiveness of thepresented NegoSI algorithm compared with other state-of-the-art MARLalgorithms.